-
Notifications
You must be signed in to change notification settings - Fork 482
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tighten up IterativeFinder async close behavior (DHT iterator continues after consumer breaks out of it) #3599
Conversation
Hello, thank you for looking into it! Since aclosing is introduced in python 3.10, I think it would be better to add the snippet that they show as equivalent to utils.py:
Then, we can use it from utils until we support 3.10, changing the import to the built-in one. I modified the snippet to test that:
Doing it this way, the pause and search messages clean, like
This looks cleaner than implementing the finally->aclose and other changes on every call and we would be able to use the built in version when it is out. |
Implemented an aclosing() in lbry/utils.py as suggested. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the quick reply!
Aside from trapping the CancelledError, I think this looks great but we need to separate the change in behavior from the iterator bugfix to ease testing and changelog.
I think I have addressed the 3 comments. The IterativeFinder is mostly back to the original logic with no generator function stuff or output limiting based on max_results. Let me know if there are more comments or a request to squash changes. |
I've seen this failure in my runs as well. Seems like the upper bound needs a bump analogous to: File "/home/runner/work/lbry-sdk/lbry-sdk/tests/integration/other/test_exchange_rate_manager.py", line 30, in test_exchange_rate_manager |
Argggggh :-/ test_exchange_rate_manager.py still failing, but due to server being down:
|
@moodyjon Can you please rebase on master instead of merging? Using rebase you can also remove the merge commits, making it a cleaner history |
c8f1aa1
to
f91bf8f
Compare
I had been observing/merging the test changes. Added one more test change yesterday: f91bf8f
OK, I think I have rebased correctly. |
…ait after task cancel(). Also make IterativeFinder a proper AsyncGenerator. This gives it an offically recognized aclose() method and could help with clean finalization.
…r exception is received, which is unhelpful, and complicates shutting down the generator.
…emove any new places enforcing max_results.
f91bf8f
to
e5e9873
Compare
Removed exchange rate test bump of upper limit. It was changed in 5c708e1 |
@moodyjon thank you very much! |
See: #3504
Script:
scratch.py.txt
Script output:
scratch_1.txt
Initially there is a lot of background activity from IterativeNodeFinder during what I presume is startup.
In steady state there is little activity after each "PAUSE---". Usually just stuff related to cancellation, if anything.
Eventually, I hope the "async with aclosing(...)" feature can be used:
https://docs.python.org/3.10/library/contextlib.html?highlight=aclosing#contextlib.aclosing